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SUMMARY 



Problem 



The Advanced Simulation in Undergraduate Hlot Training (ASUPT) facili^is d^ngiftd to be a 
research device capable of providing answers regarding the hardware design and effective use of flight 
simulators. Using stateof-the-^ motion and visual systems, the relationship betweei^^slmulator fidelity and 
training effectiveness, as well as the applicability of advanced training concepts ape to be investigated. Smce 
ASUPT was desigrved to be a research simulator, the development of an adec^uate perforAiance measurement 
system becomes- the foundation of the proposed program of research. . . ' • 

^ Approach 

One of the salient characteristics of flying is that it is criterion -directed. The execution of any 
maneuver requires that a certain dfefinable objective be met. The degree to which these objectives are met 
would appear to represent an adequate description of performance. In other words, a cnterion-referenced 
approach to .objective performance measurement is proposed. Consequently, for each maneuver, the 
criterion objectives must be defined in terms of parameters available within the simulator.. Using this ^ 
approach, a set of performance measures can be generated for each of the maneuvers to be flown m 
ASUPT. * ' ' 

Results 

To evaluate the potential' of the proposed approach, two preliminary studies v/eip co 
Mcasu|tfs were generated for seven basic mstrument maneuvers of varying levels of difiiciity^J?8&ts of 
-different experience leveb flew these maneuvers and were evaluated by experienced Instructjjr^ots. The 
Jesuits indicated that. (Jj) instructor pilots were consistent in their stibjective evaluations^ the objective 

^measures correlated highly with the sjubjective evaluations, and (3) the objective rijeasures discnminated 

. between pilots of different expcriencelevels. 





Implix^ons , *' 

The data suggest the approach taken to iJe a viable mie. The basic assumptions erf the measurement 
schjpme wi^re corroborated by the data. Thp results. (1) sl^ggest instructor evaluations, represent a useful 
criterion for developj^g objective measure, (2) indicate the objectively derived Measures possess a high 
degree of validity, and (3) provide some insight* into the ma^er in which instructors assign grades. 
Potentially fruitful areas of further research are discussed. 
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rOMATED OBJECTIVE PERFORMANCE 
.NjipASUREMENr SYSTEM 

/ ' ^ ' ' 

. L INTRODUCnON , 
■ 1- . ■ ' 

Th^ Advanced .Simulation in Undergraduate Pilot Training (ASUPT) facility is designed to be a 
researcn device capable pf providing ans^YfijSHfc^rding both the hardware design and effective use of flight 
sijnulators. Using state-of-the-art motion and visual systems, the relationship between simulator fidelity and 
the training effectiveness of these systems, as well as the applicability, of advanced training concepts are to 
be investigated. Since ASUPT is desired to be a research Sffiulator, the development of an adequate 
performance measurement system becomes an essential component of the research program. This report 
documents 'the approach to performance measurement system' development which has been taken and 
presents the results of two brief validation studies. , . \ 

Measurement System Requirements 

The criterion for evaluating a flight simulation device is its training effectiveness. From past evidence 
indicating positi^ transfer effects, it is assumed that performance in the simulator will be positively related 
to 'performance in the aircraft. As a result, the thrust of the present effort is to develop measures which 
reflect performance in the simulator. In addition, it is possible to use pilot perfonnance in the simulator as 
a criterion a|ainst which to investigate alternative simulator harawdre configurations and training strategies. 

One of the salient characteristics of flying is tljat it is bn^erion-directed. For the execution of any 
maneuver or sequence of maneuvers there are definable objectives which must be accomplished. The degree 
to which these objectives are met represents an adequate descnption of performance. In other words, a 
criterion-referenced approach ^to pilot performance forms the basis jfpr the present effort. Following such an 
approach, it is apparent that the definition of critwfon objectives is of foremost importance. Within the 
context of measurement development for ASUPT,^e critical question is whether these criterion objectives 
can be stated in terms of parameters availablp^ within the simulator. In other words, can behavioral 
objectives be defined in terms of the. state of tl>e simulated aircraft and controltoputs of the pilot? 

* Aside from the requirement to define performance in terms of observable behavioral objectives, there 
arc other constraints which are applicable.^The first is parsimony in the selection of simulator parameters to 
be sampled. Cnterion objectives are to be defined using as few parameters as possible. Furthermore, they 
are to be sampled and analyzed on a real-time basis so that the resulting measurement and feedback are 
immediate. Since measurement will be an integral part of training students in ASUPT, it is also necessary 
that the resulting output be meaningful and easily interpreted by both mstructorS and students. * 

In summary, a criterion-referenced approach to measurement system development is to be pursued 
within the consttaints of the following requirements: / 

1 . Measures will assess the degree to which the criterion-objectives are met. 

2t Measures will reflect only the most salient characteristics of performance. 

% Measures will be meaningful and interpretable to the user-the student and instructor pilot 

41 Measures will be generated.on a real-time basis scUhat feedback is immediate. 

a 




Mcasi^rriihcnt Development for ASUPT 

Afthou;gh the criterion-referenced approach represents the rationale for present developmental 
effprts, ^.further spt of assumptions have guided implementation pii ASUPT. 

\ /^Measurement system development parallels skill acquisitiojuti^he student pilot. Since student 
pilots acquire fl>ing sliills in a hierarchical fasjilon, measurement development should proceed in a similar 
manner/Slmply stated, a building block approach is assumed in \^ch measurements for basic flying skill* 
are the fitst to be impkmented. ^ 

2. '^e focus of the measurement ^system is the ifidividud flight maneuvers. This level of 
measurement seems to be most consistent with present flyii^g training syllabuses. , 



3. Maneuvers can be conceptualized as integrated sequences of steady states and transitions. The 
fundamental Hight attitudes plus transitions from one attitude to another form the conceptual segments for ^ 
most maneuvers (Meyer, Laveson, Weissman, &, Eddowes, 1974). It is necessary to define behavioral 
objectives for each segment since they ye likely to differ from one to another. 

4. Maneuver performance can he evaluated by two sets of parameters-those reflecting the state of 
the aircraft and those rejecting the control inputs of the^pilot Superior performance is assumed to 
manifest three characteristics: (a) accomplishing the criterion objectives as defined by the aircraft state ' 
parameters, (b) avoiding excessive rates and acceleration forces So that the maneuver is executed smoothly, 
and (c) accomplishing these objectives with the least amount of efibrt, that is, by minimizing control' 
inputs. ' ^ 

5. The evaluation of performaiice for a given parameter involves a comparison of the obtained value 
with' some ideal value. For parameters reflecting aircraft state, the deviation from the ideal provides an 
index of error. Since the ideal is seldom attained, it is more realistic to define acceptable performance in 
terms of an empirically determined tolerance b^d about the ideal value. For parameters reflecting 
smoothness and control inputs, the adapted ideal represents a performance level characteristic of the highly 
experienced pilot. / ^ ^' 

6. The implementation of the measurement system requires four phases of development: (a) 
definition of criterion objective in terms of a candidate set of simulator parameters, (b) evaluation of the 
proposed set of measures for the purpose of validation and Amplification, (c) specification of criterion 
performance by requiring experience instructor ^pflo'ts to fly the ^aneuyer in 'question, and (d) the 
collection of nojrmative data using students as they progress througfi the. training program. * 

Preliminary Evaluatiofl - Study I ^ :::r:rr^ — :: — 



In keeping with the approach outlined previously, the first maneuver for which measures w^re 
developed was straight-and-level flight. Since the criterion pbjjsctives for thi? maneuver are 
well-defined^maintain constant altitude, airspeed and heading-it was felt that an evaluation of its 
measurement scheme would provide a good starting place to determine whether the approach held promise. 
Any performance assessment system, if it is valid, should have .at least two characteristics. First, it should 
bear a positive relationship to expert opinion regarding the qu^ty of performance. Second, i}, should 
reliably discriminate between subjects having differing levels of Experience. The demonstration of such' 
construct validity seemed necessary for the evaluation of the present measurement scheme. 

Using the pre-programming capability in ASUPT, a simple test scenario was developed. (1) initialize 
the simulator to 15,000 feet, 160 knots, heading 090 degrees^^^) unfrec/te the 'emulator and allow 10 
seconds for the pilot to "settle down," (3) sample selected parameters for one mi^iteraqd (4) freeze the 
simulator. The parameters sampled were altitude, airspeed, heading, stidc move^m, throttie movement, 
elevator stick force, pitch rate, ^>itch acceleration, roll rate, roU accelerations, vi^ical velocity, an^rtical 
acceleration. Mean deviation and root-mean-square (RMS) deviations wore com/uted for alritude, aiftpeed, 
heading, and stick force. RMS scores were computed for the remaining parameters. 

A seven.point rating form was developed for use by instructor pilots Who were to provide qualitative 
evaluation*. Five items were^ to be rated: (1) altitude control, (2) airspeed xjontrol, 0) heading control, (4)' 
over-all level of performance*, and (5) smoothness. For each evaluation, two raters were'tlsed-one inside the 
cockpit and another at the console who only observed the repeater instruments. In thft inannet, an estimate 
of inter-rater reliability could be computed. - " * 

Subjects for the initial evaluation weret .12 Air Force employees (Flying TrainingDivision, Air Force 
Human Resources Laboratory) anrf one student pilot. Piloting experience ranged from zero to4.000+ 
hours. Each subject was^^ven a two-minute practice/instruction periQd foUowed by five one-minuteTHils 
Three instructor pflots were used as raters and were randomly alternated between the cockpit and console 
M missions were flown using fuU motion and G-seat. Of the total 65 runs, one was lost due to system 
failure. ^ ' * » 

The means of the cockpit and console ratings, along with their correlations, are presented in Table 1. 
llie high inte^corrclations suggest the instructors to be highly 'consistent in theirtatings. It is interesting to 
note that the lowest correIati«)n is for smoothness. This is most likely due to availabiUty of motion dies for^ 
the rater inside the ccfckpit. In anj^case, the high interconelatiogs suggest the ratings to be a relmWe 
criterion against whiclrtcnrdidate iUe objective measures^ 



Table L Comparison if Instructor Judgments at Cockpit and ConsolcL 
For Straight and Level 



t 



Measure 



Almtide Control 
Heading Control 
Airspeed Control 
Overall Rating 
Smoothness 



Mean 

CockPit 



4.7458 

4.5424 
4.6271 

4.45T4.. 



Mean 
Console 



4.7031 

4.6562 

4.734-4^" 

4.6719 

4.6250 



inter.-$at%r 
Correlation 



.8722 • 
.9257 
.8535 ' 
.9109 
.6297 



^ The next step was to determine vsdiether the objectively derived measures would reliably predict the 
instructor ratings. The adopted criteria were the overall rating and smoothness rating. Table 2 presents the 
.correlations between the objectively derived measures of performance and these instructor evaluations for 
both the .cockpit and console raters. A glance at these results suggest several things. First, the correlations 
between the measures and instructor evaluations are fairly consistent for both the cockpit and console 
ratings. Secon4, and most importantly, substantial relationships exist between a number of the objectively , 
'derived measures and the instructor pilot (iP) subjective evaluations. Third, the measure RMS vertical 
velocity predicted both criteria quite well. These findings were highly encouraging-suggesting that these 
objectively derived measures do relate to the instructor* s evaluation of performance. 

• • Table 2. Correlations of Objective Performance Measures 

, With Instructor Evaluations for Straight and Level 



Measure 



Overall Rating 



Cockpit 



Console 



Smoothness Rating 



Cockpit 



Consofe 



Mean* Altitude Error 
* Mean Airspeed Error 

Mean Pleading Error 

RMS Altitude 

RMSAii^peed 

RMS Heading 

Mean Stick jForce 

RMS Stick Force 

RMS Stick Movement 
VRmS Throttle Movement 

RMS Rtch Rate • 

RMS Ktch Acceleration 

RMS Roll Rate 

RMS Roll Acceleration 

RMS Vertical Velocity 

RMS Vertical Acceleration 



-.6492 
.5826 
-.5075 
-.7803" 
-.7690 
-.6393 
.2740 
-.4434 
-.0098* 
-.1643* 
-.3168 
.0781* 
-.0371* 
-.1527* 
-.7737 
-.4172 



-.7-238' 
' .6515 
-..5134 
-^.8348 
-.7739 
-.6498 
.3859 
-.3101 
-.0237=^ 
-.1855* 
*-.2625 

.1277* 
-.0500* 
-.0976* 
-.7560 
-.3770- 



-.4645 
.4365 
-.4729 
-.5750 
-.5678 
-.5931 
.3717 
-.3249 
-.2789 
-.3320 
-»5019 
-.2006* 
-.0365* 
.0145* 
-.7004 
-.4663 



-.4834 
.4331 
-.525 
-.6071 
-.6193 
-.6941 
.3560 
-.2639 
-.3369 
-.3028 
-.5036 




.0931* 
.0009* 

-.7277. 

-.5194 



* Nonsigmficant. ' ' ' . 

Using a forward selection multiple regression procedure, subsets of variables were selected which were 
predictive of the criterion. An iterative procedure was used wherein variables were added to the prediiQtion 
equation until the increment in explained variance became statistically nonsignificant. At this point, the 
variables in the prediction equation were eliminated from the predictor set and the procedure repearted. In 
this manner multiple sets of predictors were defined. The criteria adopted were the overall rating and 
smoothness rating obtained from within the cockpit. Using the equations developed against the cockpit 
ratings, an attempt was made to pufdict the rajHngs obtained at the console. The results of these analyses are 
presented in Tables 3 and 4. ^ ♦ / 
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Table 3. Prediction of OVerail Instructor PUot Ratings 
X For Straight and Level 

— — ' ' - 

Mtituf Cockgit - ContoU 

Setl ' ' 

RMS Altitude .9066 ^ , - .9044 

Mean Altitude Error 

Mean Stidk Force , * 
RMS Throttle Movement 

Set 2 

RMS Vertical Velocity , .8839' ^ ^ .8867 

RMS Airspeed » 
RMS Stick Movement 
RMS Rtch Rate 

Set 3 

RMS Heading .8303 .8178 

Mean Airspeed Error 
RMS Stick Force 
RMS Pitch Acceleration 
RMS Vertical Acceleration 

, • Set4 
Mean Heading Error .5075 .5134 



Table 4. Prediction of Smoothness Ratings 
For Straight and Level . 



Mwur^ Cockpit ^ . Cont ole 

Setl 

RMS Vertical Velocity .7926 . • ,8005^ 

RMS Throttle Movement 
RMS Altitude 

RMS Pitch Rate ' / Q 

Set 2 

RMS Heading .7155 .7086 

Mean Altitude Error 

RMS Rtch Acceleration * 

Set 3 

RMS Airspeed ,6718 ^ .7053 

RMS Vertical Velocity 

Set4 ' ^ > 
Mean Heading Error .6248 .6496 

Mean Airspeed Error 
RMS Stick Movement 

Set 5 " 

Mean Stick Force ' .5585 ;5277 ' 

RMS Stick Force 



, Four subsck of vgdables were Selected which were predictive of the, overall ratings. The first sublet 
consisting of^JS altitude, mean altitude enor, mean stick force| and RMS throttle movement yielded a 
multiple K ^\9066, The correlation between the ja5cdicte4 score (using the equation developed for t^c 
cockpit rating) and the console rating was 5044. sfioilar degrees of conespondence were obtainedjfor 
remaining subsets of prcjdictions. For smoothness, five subsets of predictors were identified. Again, hign 
degrees of correspondence were obtained between the multiple Rs developed for the cockpit ratings anffWF" 
»ibsequent correlation between predicted scores arid the console ratings. 

/ The results of the first studx^re highly encouraging and seemed to wanant the^ following 
conclusions. First, instructor p^ots are highly consistent in their evaluations of performance. Consequently, 
it is posable to ust such evaluations as one criterion against which to validate objective measures of 
performance. Second, the objective measures of performance developed for straight and level will reliably 
predict instructor evaluations. The demonstration of such predictive validity suggested the approach taken 
to be a reasonable one. To further evaluate the proposed system, a second study was undertaken. 



Pitlimmary Evaluation - Study n 

Scenarios -^ere developed and the performance measurement software written for^kftj^lov^g 
maneuvers, change of airspeed, constant ^ rspee d climbs/descents, rate climbs/d^cents, and the steep^ 
Slmflar fating forms were develo^d for each maneuver and the amultaneous cockpit/console evalu^c 
J)roccdures followed. Four T-37 instructors were the raters-two alternating at the console and the oil 
two alternating in the cockpit. Ten subjects were used in Uie second study, again representing a ^ide i 
of skills. Four student pilots in T-37 training, three T-37 IPs, and three civilians were included. Each ivXytcX 
flew the following set of maneuvers: three airspeed changes; one constant airspeed dhnb; one constant 
airspeed descent, one, rate climb; one rate descent; and three steep turns. For each dimb/descent a level-off 
to ^titude was required. 

Inter-rater correlations were computed for each maneuver and are presented in table 5. As indicated,' 
the data for climbs and descents were pooled. Overall inter-rater correlations were computed for categories 
^hich were rated for all maneuvers. The data indicates substantial agreemeiit among the raters, especially 
for the overall and smoothness ratings, even though the values were somewhat less than obtained for 
straigiht-and-leveL Several possible reasons for the lowered inter-rater conelations should be mentioned. 
Rrst, the maneuvers in the second study were of increased cojnplexity. Since these maneuvers require 
several transitions in addition to a steady state condition, the instructor's job of monitoring all the relevant 
parameters is increased. likewise, the performance of Iransitioris from one steady state to another increases 
the number of cues available to the rater within the cockpit. A second-possibility concerns rater bias. It is 
possible that different subjective criteria were used in ratings of the T-37 IPs as opposed to students. An 
examination of the ratings of one of the IPs performance records yielded large discrepancies between the 
cockpit and console ratings. The objective measures appeared to agree with the cockpit ratings in th^t the 
performance was quite good. However,' according to tiie console rater, the performance was considered to 
be unsatisfactory. Such data strongly suggest the possibility of rater bias. 

TflWe J. Inter-Rater Correlations for Study n ' , . 

. . xli. ^ 1 



' Changtof CAS * Ratt SUtp 

M«asurt Airspttd' . ' Cnmb/D%scfrfr^^ ^ -^Ciimb/Pttctnt Tgrn ' OvtraU 

Altitude Control .7714 . .6334 ' .7740 .6935' .7279 

V Airsiftjed Control .7763 ' .,4818 .8078 ; '.423g .6758 

HeadingControl ' .6121 .8987 - :7534 ^ + * 

Rate Control' + -I- . ' ^ ' .6272 > • + * 

Bank Control + , + ' + .?,^43 * 

OvwaU * .7741 ^ .6822 .8045 ^ nl2\ \ .7716/ 

Smoothi^ss \7311v .7055 .9661 v .8592 .8368^ 



+ Not c^^ffnputed formancuvbt. 

^ Nbt^omputcd since item notratcd in^l maneuvers. A 



Correlations betw^n each of the objective measures and the overall and smoothness ratings for each 
maneuver v/ere computed and presented in Table 6. In this case, the ratings from the IP in thejcockpit v/ere 
used 33 the criteria. A perusal of the data warranU several conclusions. As expected, parameters reflectmg 
performance of the criterion objectives v/ere most related to the ratings, likewise, the JIMS values about 
the ideal were correlated more than mean deviations. The only exception was Riikbank error. In 
this case, an error in tiie computing software was discovered, tiiereby invalidating tiie.resultingmeasure-sc, 

A forward selection regression analysis was computed in an attempt to develop predictiokequations 
for the overall steep turn rating. The steep turn ms selected since it represented tiie most diffi^ of the 
maneuvers tested. The initial sujiwt of seven variables* selected by the procedure yielded a multjplt 
.8820' This equation was thed used to predict the console ratings. The obtained correlation t 
predicted and observed console ratings' was 5137. Again, multiple subsets were isolated whic__^. 
' predictive of the criterion. Although not verified, it seems likely tiiat dmilar sets of predicton e^^ations 
could have been developed for iht other maneuvers. In any case, it is certainly cleaUfaat tiie objective 
measures developed are highly predictive of instructor ratings, thereby demonstratmg tiie vahdity of the 
present approach to measurement 

A further set of analyses were computed in order to^refatc the objective measures to the mstructor 
evaluations. Performance records for aD maneuvers were pooled and placed into groups accordmg to the 
evaluation of the cocki^ instructor. Four groups were defined according to tiie grade assigned of U, F, G, 
dr E Descriptive statistics were tiien computed for each of the four groups. The results are presented m 
Table 7. Measures for rate of cHmb and bank were deleted due to tiie small number rfcases withm each of 
the groups. An examination of the data warrants several conclusions. Firsts it is apparent that there are 
clearly defined trends for a number of tiie objective measures across the four grpups. There are clear cut 
decreases m root mean square for airspeed,* heading, and altitude as a function, of tiie sujbjective ratings. 
Likewise, there are dei^reases in the variability aaoss subjects. Consequently, lowered ratings are 
characterized by increasing within-subject enor as well as. increasing between^bject vanabiUty. Such data 
reflect the fdci that there is one way to execute the maneuver correctly, but many ways to coipmit OTors 
and therefore receive a lower evaluation. 

The results also Verified the ' assumption that superior maneijver 'performance involves the 
minimization of control inputs. Performances rated excellent were those which minimized the amount of 
stick movements .and also minimized stick deviation from the nun position. Furthermore, supenor 
p^ormai\(^ was ajso characterized.^y miriimum ampunt of control force, or, in other wi^ds, tiie efficient 
use of trim: WhjJe stick inputs werf found to be related to rated performance, throttie movements were 
not Of the proposed measures of smoothness only pitch rate, vertical velocity and vertical acceleration 
were related to rated performance. Ag^, the pattern ©fdecreasmg means and variances was found. 

As discusied previoudy , a requirement of tiie measurement system is tiiat it reliably discriradfe^ 
between pilots of different experience levels. In. this case, depending on tiieir previous flying expenrace, 
subjects 'were pimped into one of two cate^orie^-high versus low e^jience-wid descriptive statistie^ 
conoputed. The results are presented in Tabl? 8 It should be pointed o.|A'Tt^fjp^e statistics computed for^'^ 
. each objective measure were based on data rejUlting from all tiie iraiheuVers in which that measure was 
computed For example, the statistics for heading measure^ were based on all maneuvers except the steep 
tum. The results Indicate that a substantial number of the objective measures will differentiate between the 
twa groups. Generally, the inexperienced group was characterized by higher error scores and much greater 
variabiHty. For the steady state parameters, the RMS error scores were more discriminative tiian tiie mean 
error scores. 

^fajpBcations ^ ' * ; ^ , 

The lesults of these two evahiation studies warrant a number of conduaons. Pint, the data suggest 
the approach taken to be a viable one. The assumptions conceming superior maneuver performance have 
been corroborated by Uie data. Experienced pilots, and likewise tiipse performances rated excellent, were ' 
' characterized hf (}) meeting tiie crifericm objectives in terms of minimizing aircraft state errore, (2) 
minimizing rates and accclerations-at least 'in the mch and Z axes, and (3) minimizing control mputs. In 
otiier words, superior performance involves getting the job done, doing it smootiily, and witii a least 
amount of effort. .. . , . 
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Table & Comparison of Objective Measures for Experienced 
And Inexperience Mots 
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Second, the data indicate that instructor evaluations are a ifeftle criterion for future measurement 
system development' the^ relatively high levels of agreement between the cockpit and console ratings are 
particularly encouraging since the availability of cues was radically different for each rater. The console 
rater only had access to the repeater instruments while the instructor in the cockpit could observe the 
students* behavior in' addition to the flight instruments. Furthermore, kinesthetic cues were also available to 
the cockpit rater. The importance of these different cue sources in instructor evaluations is an area which 
should be addressed in future research studies. 

Third, the objectively derived measures were shown to possess a certain degree of validity. Significant 
correlations between these measures and instructor evaluations were obtained. Furthermore, the measures 
were shown to discriminate between, pilots , pf different experience levels. Th«e results are most 
^ encoura^g and convincingly demonstrate the fruitfulness of the present approach. 

Fourth, the results provide some insight into the manner in whiqh instructors assign grades. A 
hierarchical model seems most cons^tent wHh the data, To receive an excellent (E) rating, errors on the 
critical parameters must aD be low. As* the quality of the rating decreases^ the potential for different errors 
increase. In fact, it is posable to commit error involving one parameter, control tiie otiiers quite well, and 
still receive a low evaluation. This suggests that both the mmiber and degree of error are important. The 
investigation of instructors grading strate^es is another prime, area for future research. 

Fifth, tiie results provide preliminary data concerning tiie simplification of tiie present set of 
parameters. As expected, paraipeters reflecting performance of the criterion objectives yielded tiie highest 
validity coefficients. However, measures of mean error were not as effective as toot-mean-square error 
Furthermore, a number of tiie proposed measures of smoothness did not produce any significant 
relationships. Roll rate, rdl acceleration, and pitch acceleration were not related to eitiier instructor rating 
or cx^rience level Lik^e/throttie movements were not found to be important. 
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Taken as ^ whole, the results of these preliqiinaiy investigatiQjis are cocouxaging. The demonstrated 
validity of measures, developed for these basic instrument maneuven, indicate the fruitftdness of the 
present ap prnfl<;?h Effnm ^r^* r"rr<^ntly 4inderway for thc^ development of performance measu^ei-fw-morc 



complex maneuvers. It is expected that the resulting measurement system will meet both the research and 
training needs for future studies to be accomplished in ASUPT. 
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